System and Method for Patient Monitoring

ABSTRACT

The present disclosure provides a system and method for monitoring the cognitive state of a patient based on eye image data. The patient monitoring system comprising a camera unit configured for recording images of an eye of the patient, and a data processing sub-system in data communication with the camera and being operable to (i) receive and process eye image data from said camera, (ii) classify said eye image data into gestures and identify such gestures indicative of the cognitive state of the patient, and (iii) transmit a signal communicating said cognitive state to a remote unit. The system may further comprise an actuator module and an output unit wherein said output may be an automated medical questionnaire.

TECHNOLOGICAL FIELD

The present disclosure relates to a system that allows the evaluation ofa patient's cognitive state based on the patient's eye image data.

BACKGROUND

Lack of communication is a common, unmet need of hospitalized patients,specifically in intensive care units. It is believed that effectivecommunication deprivation could be a contributing factor to thedevelopment of intensive care unit (ICU) delirium. Effectivecommunication could be part of prevention and treatment for ICU-deliriumin patients. The current standard of communication for the criticallyill mechanically ventilated patients, for example, is limited tonodding, writing, and pointing at communication boards.

System and method for enabling user communication by tracking the eye ofa user are known.

WO2016142933 discloses such a system with a selection interface thatselectively presents a series of communication options to the user. Alight sensor detects light reflected from the eye of the user andprovides a correlative signal, which is processed to determine arelative eye orientation with respect to the head of the user. Based onthe determined relative eye orientation, a selected communication optionis determined and implemented.

WO2019111257, which is incorporated herein by reference in its entirety,discloses a control system that interfaces with an individual throughtracking the eyes and/or tracking other physiological signals generatedby an individual. The system is configured to classify the captured eyeimages into gestures, that emulate a joystick-like control of thecomputer. These gestures permit the user to operate, for instance, acomputer or a system with menu items.

GENERAL DESCRIPTION

A wearable device may serve as a monitoring device, inter alia as anemergency-call device, and to improve patients' orientation in general.In addition, a wearable device could allow broader and more effectivecommunication relative to the current standard of communication forintensive care unit (ICU) patients.

The system of this disclosure is useful for identifying or monitoring acognitive state of patients that have lost, temporarily or permanently,their ability to communicate in a verbal manner and may be in varyingcognitive states. Relevant patient populations for the system of thisdisclosure are patients whose relevant motor functionalities areimpaired. The impairment may be temporary, e.g. in the case of patientin an ICU or patients recovering from trauma (e.g. an accident); or as aresult of a permanent morbidity such as paralysis caused by a central orperipheral nervous system disease (e.g. ALS).

The present disclosure concerns a system comprising a camera thatcaptures a sequence of image data of one or both of the patient's eyes.The eye image data is received and processed by a designated dataprocessing sub-system which is operable to classify said eye image dataand interpret it into eye gesture, such as eye opening or a blink. Thedata processing sub-system next identifies, among the resulted eyegestures, unique gestures, gestures patterns, and sequence of gesturesthat characterize a cognitive state of interest such as awakeness ordelirium, based on defined criteria (the criteria may be either fixed ordynamic criteria, for example a criteria which varies across medicaldisciplines or geographical location, or a criteria derived from amedical textbook). The data processing sub-system may determine thatsaid patient is at, under, or likely to be at or under a cognitive statebased on said identification. Upon identification of a cognitive stateof interest the sub-system will report the identification to a remoteunit by transmitting a signal. For example, for a patient hospitalizedin an ICU, who just woke up, as indicated by eyes opening for the firsttime in days, the system will send an alert signal to any one or acombination of a connected nurse unit, medical staff member, or a familymember. The system disclosed herein may also be implemented in anymedical or non-medical institution, including but not limited torehabilitation facilities, nursing facilities, a long-term acute carefacilities and senior citizen housing.

By some embodiments the system comprising a wearable headset, whereinthe camera is carried on a head unit configured for fitting onto apatient's head.

The system may also provide any output including patient-selectedcontent, medical questionnaire and feedback to the patient, as thesystem may further comprise an actuator module configured to drive anoutput unit to present an output to the patient. For example, the systemwould allow the automated computerized identification of a deliriumstate in an intensive care patient. Specifically, the system wouldprovide a computerized interface for the well-established ConfusionAssessment Method for the ICU (CAM-ICU) test to the patient, who wouldbe able to provide a response via the system (that is via thecommunication signals the system offers). The system may be configuredto receive and process additional physiological data other than eyeimage data to facilitate the identification of the cognitive state. Bysome embodiments, the system would also be configured receive andprocess an additional data such as a verbal input from the user.

The system may, by some embodiments, have a high accuracy rate and briefinference time to allow a simpler yet highly reliable communicationsystem for individuals who's ability to communicate is compromised. Thiscould result in, inter alia, shorter training sessions and a moreefficient use of Graphics Processing Unit (GPU) or, by some embodimentsfaster data transmission to a remote processor (server) which processesand analyzes the transmitted data more efficiently.

According to a first of its aspects, there is provided a patientmonitoring system for identifying a cognitive state of a patient, thesystem comprising a camera configured for recording images of an eye ofthe patient; a data processing sub-system in data communication with thecamera and being operable to (i) receive and process eye image data fromsaid camera; (ii) classify said eye image data into gestures andidentifying such gestures indicative of the cognitive state of thepatient, and (iii) transmit a signal communicating said cognitive stateto a remote unit. The system is developed and designed to monitor apatient. This may include for example any hospitalized patient such asintensive care unit patient, ALS patient, a locked-in patient, amechanically ventilated patient, a critically ill patient and a patientwithout an ability to communicate verbally. Additional examples arenon-hospitalized patients under medical or caregiver supervision.

The term monitoring encompasses a continuous or intermitted monitoring.By some embodiments said monitoring is seconds or minutes-long,monitoring session for the purpose of medical evaluation. By otherembodiments the monitoring is an extended monitoring for example forpatients that are hospitalized for days, weeks, months and years.

The term identifying (or its derivations) refers to any binarydetermination (such as awake or a non-awake, in pain or not, in deliriumor not), quantitate determination (for example duration of wakefulness,the amount of a daily wakefulness periods, confusion or pain level at anumeric scale of 1-10, or a likelihood index) or qualitativedetermination (such as relative sleep quality, disoriented state forinstance in relation to yesterday's comparable state) of a cognitivestate. The identification may be based on timing, sequence, duration,pattern, or other measures of the patient's eye gestures. Theidentification encompasses the likelihood, occurrence and duration atany scale (minutes, hours or days) of the cognitive state. Theidentification may be deduced from the gestures based on pre-definedcriteria. The system may also have the options of input by a physicianor other caregivers to define definitions or identification rules.

By some embodiment the identification is a predictive identification,encompassing the likelihood of a patient to manifest the cognitivestate.

By some embodiment the identification is performed based on a database,gathered based on the patient's own eye gestures or the eye gestures ofa group of patients. Said database may be based on historical or realtime eye gestures data.

The term cognitive state encompasses wakefulness, sleep state, delirium,cognitive abnormality such as decline, confusion, disorientation,abnormal attention or consciousness, impaired memory, distress, abnormaldecision making, frustration, discomfort, pain, and depression. Thecognitive state may be the patient natural cognitive state or acognitive state induced, affected, or modulated by a medicalintervention, such as a drug.

By some embodiments the patient monitoring system is screen independent.

By some embodiments the head unit is lightweight head mount, fitted ontothe patient's head by a family member, a caregiver or the patient'sitself, and may further include a bone conduction speaker\headphone. Thehead unit may be easily removed. A camera may be carried on said headunit and is configured for recording eye image data that may includeimages of any one of the eyes, both eyes, any eyelid, or both eyelids ofthe patient and generating image data representative thereof.

By some embodiments the camera is carried on a head unit configured forfitting onto a patient's head.

By some embodiments, the camera may also be mounted on a frame in thevicinity of the user, e.g. a frame of a bed, a frame that carriesmedical instruments, etc.

By some embodiments, the camera is fixed relative to the patient's eyes.

By some embodiments, the camera is not fixed relative to the patient'seyes.

By some embodiments, the camera is an infrared camera or a visible lightcamera.

Typically, operation of the system of this disclosure is independent onthe lighting conditions.

By some embodiment the position of the camera (for example as it isattached to the heat unit) is fixed relative to the patients' eye andserves as the only reference point for the captured image data.

By some embodiments, the patient eyes-based communication resemblesjoystick-like control, rather than detecting the exact location orposition the patient is looking at or at a corneal reflection (relativeto a screen for example). Also, according to this disclosure there is,typically, no need for any calibration procedure using a screen prior touse, and in fact, there is no need to use screen at all to communicateusing the system.

The term joystick-like control as described herein refers to gesturesclassification comprising tracking the position of the pupil area withinan eye image data.

By some embodiments, the joystick-like control is according to thedescription of WO2019111257, which is incorporated herein by referencein its entirety.

By some embodiments tracking the position of the pupil area is carriedout independently of the patients' face.

By some embodiments tracking the position of the pupil area is carriedout relative to a fixed camera.

By some embodiments tracking the position of the pupil area is carriedout relative to a non-fixed camera.

The pupil area in the context of this disclosure, is the pupil or anyportion thereof, identified as indicative of the pupil.

By some embodiments the position of the pupil area is determined basedon a databased comprising image data with labeled pupil or eye gestures.Said image data may be acquired from the patient itself or any otherpatient or group of patients. By some embodiments, the position of thepupil area based on said labeled databased is determined by utilizingmachine learning technique, for instance a model considering thelikelihood of a given image data to correspond to a particular gesture.

By some embodiments the position of the pupil area may be determinedbased on its position within a threshold map, wherein a particularposition is determined whenever the pupil area touches a border of, ortangent to a border of, the threshold map. For instance, when the pupilarea touches the upper border of the threshold map the image data wouldbe classified as an “up” gesture, or when the pupil area is not touchingany border of the threshold map the image data would be classified as a“straight” gesture. The threshold map may be derived from a positionmap, including a region which is within the motion range of the pupilarea. By one example, the position map is defined as a rectangle definedby the upper, lower, leftmost and rightmost positions of the pupil area.By some embodiments, the threshold map covers at least one area limitedby a border that is at least 20%, 40%, 60%, 80%, 90%, 95% away from thecenter of the position map. The threshold map is typically at least 80%away from the center of the position map. The position map may beobtained based on the patients' image data or any database comprisingimage data with or without labeled gestures. Optionally, the positionmap is within a larger, region of interest (ROI), defined based onanatomical features of the eye or its surrounding.

At times, the position of the pupil area may be determined based on itsposition within a threshold map comprising more than one key zone. Thekey zones may be separate, optionally overlapping subregions within theeye image data. At times, a pupil position would be classified as agesture when the pupil area is so positioned such that it touches aborder of, tangent to a border of, or comprised within at least one keyzone. The image may be captured at a frame rate of at least 30 Hz, usinga camera with a shutter speed of at least 1/30 sec. By some embodiments,a pupil position will be classified as a gesture if the pupil maintainsits gesture-defining position for at least a defined time period, forexample 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 1 or 2 seconds.

By some embodiments, the position of the pupil area is determined byemploying computer vision tools, optionally based on circular, curved,and linear features. Optionally, the pupil area identification iscarried out based on identifying dark or black pixels in the image data.By some embodiment the center of the pupil is calculated to serves as areference based on which the eye gestures are classified. By someembodiments the eyes motion range relative to the calculated pupilcenter is also calculated to optionally serve as a reference for the“middle” eye gesture.

By some embodiments, the position map is a nonconsecutive map or a mapcomprising more than one area or a zone of an eye image.

By some embodiments the eye gesture is classified based on the imagedata of a single image data frame. By other embodiments, the eye gestureis classified based on a plurality of image data frames, optionally 2,5, 10, 20, 50 image data frames. By other embodiments the gesture isclassified based on 60%, 70%, 80%, 85%, 90%, 95% or 97% out of theplurality of image data frames within a defined time window or a definednumber of image data frames.

By some embodiments the position of the pupil area is determined byemploying object detection tools. Optionally the object detection iscombined with a deep machine learning model to identify the eye, thepupil and features thereof. Optionally the identified eye features areselected from the eye's iris, outer (limbus) circles or the iris, pupil,inner pupil region, outer pupil region, upper eyelid, lower eyelid, eyecorners or any combination thereof.

Optionally, the image data is classified into eye gestures based on anyone or combination of the eye features. Said eye features may beselected from eye's iris, pupil, inner pupil region, outer pupil region,outer (limbus) circles or the iris, upper eyelid, lower eyelid, and eyecorners.

By some embodiments said eye identification or eye features are derivedfrom the bounding box object detection. The term bounding box may relateto an area defined by two longitudes and two latitudes. Optionally, thelatitude is a decimal number in the range of −90 to 90 and the longitudeis a decimal number in the range of −180 to 180. By some embodiments thedetection is carried out based on labelled data comprising bounding boxcoordinates and image data labels.

By some embodiments the position of the pupil area is determined byemploying machine learning tools or a combination of computer vision andmachine learning tools.

Optionally, the combined computer vision and machine learning model isbased on a Single Shot Detector (SSD) algorithm.

Optionally, the combined computer vision and machine learning model isbased on a You only Look Once (YOLO) algorithm. The YOLO algorithm mayinclude two steps. A first step comprises employing an object detectionalgorithm to detect at least 2, 3, 4 or 5 key zones within the eye imagedata. A second step comprises employing a supervised machine learningalgorithm to determine the eye pupil within the eye image data, based onthe coordinates of said key zones. By some embodiments the position ofthe pupil area is determined based on a machine learning model.Optionally, said model is based on the patient's image data. Optionally,said model is based on a group of patient's image data. Optionally saidmodel is based on a group of healthy individuals' image data.

Optionally, said model is a supervised, semi-supervised or unsupervisedmodel.

By some embodiments said supervised model is based on manually labeledor automatically labeled image data.

By some embodiments the boundaries between different gestures aredefined based on the patients' image data.

By some embodiments, gesture classification is based on employingmachine learning techniques. Specifically, the machine learning modelmay be a neural networks model consisting multiple lineartransformations layers and subsequent element-wise nonlinearities. Theclassification may comprise eye characterization of an individualpatient or across patients. By some embodiments the classificationestimates the range of eye motion. The machine learning model may employa classifier selected from: logistic regressions, support vector machine(SVM), or a random forest

By some embodiments said model is a deep leaning model, optionally aconvolutional neural network (CNN) model. By some embodiments said modelclassifies image data into at least 5 basic gestures. Optionally saidbasic gestures are selected from blink, up, down, left, right, middle(straight), diagonal upper left, diagonal upper right, diagonal lowerleft, and diagonal lower right.

By some embodiments said gesture classification is at an accuracy of atleast 80%, 85%, 90%, 95%, 97%, 98, or 99%.

By some embodiments said gesture classification average inference timeis up to 100 ms, 125 ms, 150 ms, 175 ms, 200 ms, 240 ms, 250 ms, 260 ms,270 ms, 300 m, 350 ms, or any number in between. By some embodimentssaid gesture classification average inference time is in the range of100-200 ms, e.g. 125±25 ms. Said inference time may correspond to 3-10frames per second (fps), at times 8 fps.

By some embodiments, a gesture would be classified based on the majorityof gestures within a pre-determined time window. A basic time window maybe defined (e.g. in the range of 50-200 ms) and the gesture beingclassified in several consecutive such time windows and then defining asa gesture is when the same classification occurs in the majority of thetime windows. For example, in case a 125 ms window is applied, thegesture would be determined every 375 ms (namely after 3 such windows)if the same gesture were classified in 2 out of 3 images. This has theadvantage of avoiding hypersensitive response, thus allowing a morestable output.

By some embodiments the machine learning model is further used forheadset placement, optionally by classifying whether at least one eye iswithin the image frame. Optionally the image data is classified into 3classes, being at least one eye entirely within the frame, no eyeswithin the frame, and at least one eye centered within the frame.

By some embodiments the machine learning model is first used for headsetplacements by identifying at least one eye location within the imagedata. Next, gesture classification is carried out by employing acombination of computer vision and machine learning tools.

By some embodiments a data processing sub-system is a distributed ornon-distributed, parallel sub-system.

By some embodiments the classification of said image data into gesturesand the identification of such gestures indicative of the cognitivestate of the patient is carried out by distributed sub-systemcomponents.

By some embodiments a data processing sub-system is in datacommunication, optionally wireless communication, with the camera andbeing operable to receive and process image data from said camera, andto further classify said image data into gestures (optionally theclassification is carried out according to the joystick-like controldescribed above). Said gestures may comprise voluntary or involuntaryeye gesture. Said gestures may comprise straight, center, right, left,up, down, diagonal upper left, diagonal upper right, diagonal lowerleft, and diagonal lower right positions of the pupil, any sequence ofpupil positions, eye closing, eye opening, a curved eye movement, eyemovement behind closed eyelid (for example rapid eye movement duringsleep), increase or decrease of pupil size (e.g. dilated or constrictedpupil), expansion or reduction of part or inner part of the pupil eyelidtwitching, blinks and a sequence of eyelid blinks. Said gestures mayalso include any eye or eyelid movement that is associated with eyeliddisorders such as ptosis, eyelid retraction, decreased or increased,blinking and eyelid apraxia.

Optionally, the gestures relate to one of the eyes, or both eyes.

Optionally, the gestures comprise a sequence of 2 or more eyelid blinks.

The gesture may selected from any one or combination of eye gesturesknown in the art, for example the gesture may be a fixation (stationarygesture or gaze) or a series of fixations and their durations, gesturesor gaze points and clusters and distributions thereof.

By some embodiments the system asks the patient to perform a straightgesture in between other gesture.

By some embodiments the blink gesture is identified as a region of darkpixels, or by employing an artificial intelligence model, optionally amachine learning model that classifies the image data of the eye asimage of a closed eye (either via supervised or non-supervised leaning).

By some embodiments the eye opening gesture is identified as a region ofthe pupil, or by employing an artificial intelligence model, optionallya machine learning model that classifies the image data of the eye asimage of an open eye or not closed eye.

By some embodiments, the eye closing gesture is identified as a sequenceof image data frames of a closed eye after at least one image of notclosed eye.

By some embodiments the rapid eye movement (REM) gesture (typical to REMsleep) identified as a sequence of image data frames with rapid changesbetween them. As a non-limiting example, a sequence of gesture“down-middle-down-middle-down-middle” will serve as an image dataclassified as a REM gesture.

By some embodiments a gesture would be classified when the pupil areatouches a border of, tangent to a border of, or comprises within thethreshold map. The image may be captured at a frame rate of at least 30Hz, using a camera with a shutter speed of at least 1/30 sec. By someembodiments, a pupil position will be classified as a gesture if thepupil maintains its gesture-defining position for at least a definedtime period, for example 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 1or 2 seconds.

By some embodiments any one of the type of gestures, number of gestures,the gestures duration, and the corresponding signals and outputs aredefined by the patient or a caregiver.

By some embodiments a signal is transmitted based on a single gesture, aplurality of gesture or a sequence of gestures. Optionally a signal istransmitted based on the timing, sequence, duration, pattern, othermeasures of the patient's eye gestures and any combination thereof.

By some embodiments the eye-opening gesture for at least 1, 3, 5, 7, 10,30, 60 seconds initiates an “awake” signal.

By some embodiments the eye closing gesture for at least 1, 3, 5, 7, 10,30, or 60 seconds initiates an “asleep” signal.

By some embodiments a series of 1, 2, 3 or 5 blinks may initiate a “callfor help” signal.

By some embodiments, a series of up to 10 blinks, within up to 30seconds selects a signal.

By some embodiments opening or closing at least one of the eyes for 1,2, 3, 4, 5, 6, 7, 8, 9, 10, or 30 seconds sends a signal. Optionally,said signal is a rest mode single to the system.

By some embodiments, the signal reports to a caregiver the awakensperiods of the patients over a 24 hours period.

By some embodiments, the signal is a recommendation provided by thesystem regarding the timing of physically visiting the patient' room.

According to another non-limiting embodiments, a patient with a limitedability may operate the system using a single gesture, optionally basedon his definitions or eye motion ranges, for instance only a “left”gesture.

By some embodiments the gestures permitting the patient to operate thecomputer are general viewing at a defined direction (eye gesture),instead of a gesture in which the patient is looking at a particularlocation (eye gaze). For example, a general left stare may serve as agesture, even if the patient is not focusing his sight at a specificphysical or virtual object.

By some embodiments the identification of gestures indicative of thecognitive state of the patient is carried out based on rules derivedfrom the general medical knowledge and methods known in the art, orbased on criteria defined by a nurse, a physician, a caregiver or thepatient. Said rules can be either pre-defined or derived after aninitial monitoring of an individual patient. Said rules implemented withregards to a patient can be derived from the patient himself or from atypical population group. By some embodiments the identification rule isa fixed rule or rules (based on solid medical knowledge) or dynamic rule(for example based on a single edition of a medical textbook, or a rulethat varies across medical disciplines, geographical regions etc.). Bysome embodiments the rules are dynamic in the sense they are derivedbased on online artificial intelligence algorithms Said identificationmay be carried out as part of monitoring the status of the patient(either during a given time point or in an ongoing manner) or forpredicting a cognitive state. said predictive identification,encompassing the likelihood of a patient to manifest the cognitivestate.

The cognitive state may be determined in combination with anyphysiological or non-physiological data. By some embodiments, thecognitive state in determined based on any combination of eye image dataand physiological data received by the system. By some embodiments thecognitive state in determined based on any combination eye image data,physiological data, and additional data received by the system. Saidadditional data may be any quantitative, qualitative, probabilistic,textual, personal or populational data. Said data may be received onlineor at a pre-defined time point. Said data may originate from thepatient, medical staff member, or family member.

Last, once a cognitive state is identified by the system, the dataprocessing sub-system transmit a signal associated with the saidcognitive state to a remote unit.

By some embodiments said signal may be any signal, optionally adigitized, computerized signal (for example to put the system into restmode, or to operate an Internet-of Things (IOT) device), or optionallyvisual signal (change of light from green to red or vice versa),auditory signal (such as an alert sound), or other digital signals suchas a text via text messaging service.

By some embodiments the signal is a word, symbol or sentence, optionallypre-defined by the patient or a caregiver or composed by the patient,optionally using a menu with letters selecting interface.

By some embodiments said remote unit is an alert unit of an intensivecare unit, a nurse unit, or a device carried by a caregiver.

By some embodiments the signal to a remote unit is transmitted viawireless communication. The term wireless communication, as used hereinmay include any form of communication that is independent of anelectrical conductor connection, such as Wi-Fi Communication, mobilecommunication systems, Bluetooth communication, infrared communication,and radio communication. By some embodiments the wireless communicationis via a wireless network such as wireless personal area network (WPAN)or a wireless body area network (WBAN).

As a non-limiting example, the signal to a remote unit may correspondsto triggering a device or an equipment in the patient's room, such asturning on or dimming the light in the room, activating a light therapysession, playing media content such as sounds or music, or controllingthe room temperature.

As a non-limiting example, the signal to a remote unit may be a signalactivating a system for improving or alleviating the patient's medical,emotional, or cognitive state (for example a system for reducingdelirium).

By some embodiments the opening of at least one eye image data is beingclassified as an open eye gesture indicative of a wakeful state, and analert signal is transmitted to a nurse unit.

By some embodiments the patient monitoring system further comprises anactuator module configured to drive an output unit to present an outputto the patient, and wherein the data processing sub-system recordsreactive image data representative of a reactive eye movement thatfollows said output, classify said reactive image data into reactivegestures and identifying such gestures indicative of the cognitive stateof the patient.

The term output encompasses any sensory output or a content to thepatient.

By some embodiments, the output is provided via an Internet-of-Things(IOT) device, such as a smart home device.

By some embodiments, the output is associated with the prevention orreduction a cognitive state, optionally ICU-delirium.

By some embodiments, the output is associated with induction orfacilitation of a cognitive state such as a sleep state.

By some embodiments, on top of an automatic actuation described above(from the recording of the image data up to a cognitive state signaltransmission), the output may be initiated in response to an eye gestureor a signal. In addition, the output may be selected based on thecognitive state automatically identified by the system, therebyoptionally positively affecting the patient cognitive state. Forexample, a relaxing music, family voices, white noise, or cognitiveexercise may be selected in response to the identification of an anxiouscognitive state.

By some embodiments the sensory output is visual (for example a messageor a question on a screen), auditory (such as a vocal instruction or aquestion), a tactile output (such as a touch stimulus across thepatients' foot) or any combination thereof.

By some embodiments the term content is any content either generic, orpersonally generated content. These may include a generic medicalinformation video, or a message delivered to the patient by hisphysician, nurse or caregiver.

By some embodiments the content is any media content selected by thepatient.

By some embodiments the media content is selected in order to better thepatients' functional performance or a medical indication such as tolower stress.

By some embodiments the media content is any visual or auditory contentfamiliar to the patient such as voices of family members or knownenvironment. The auditory content may be a pre-recorded media file ortransmitted online. By one embodiment, the content is a menu system thatpermits the patient to navigate through a menu that is presented to thepatient and controlled through his eye gestures by selecting menu itemsusing eye gestures. The presentation of the menu may be an audiblepresentation (by means of a loudspeaker, earphones, headphones,implanted audible device, etc.) or a visual presentation (through adisplay on a screen, a small display in front the patient, etc.). Themenu may be hierarchical, meaning that a selection of a menu item mayopen other, lower hierarchy selectable options.

By some embodiments, the menu is according to the description ofWO2019111257, which is incorporated herein by reference in its entirety.

By some embodiments the output is a human, digital or automatic medicalquestionnaire. At times, the type and content of the medicalquestionnaire is determined based on the patients' response to aprevious question. At times, the type of medical questionnaire, and thequestions of said medical questionnaire are determined based on reactivegestures. By one embodiment said medical questionnaire is ConfusionAssessment Method for the ICU (CAM-ICU). By one embodiment the openingof at least one eye gesture initiates the CAM-ICU output.

By some embodiments the output is a visual or audio indicating to thepatient the day, time and location of the patient.

By some embodiments the output instructs the user how to respond(optionally to said medical questionnaire) using his eye gestures.

The term reactive image data, refers to image data recorded in responseto an output provided to the patient.

The term reactive eye movement refers to eye movement recorded inresponse to an output provided to the patient.

The term reactive eye gesture refers to eye gesture classified based ona reactive eye movement or reactive image data. In some embodiments thereactive gestures are indicative of the cognitive state of the patient.

By one embodiment the opening of at least one eye gesture initiate theCAM-ICU output, and the reactive eye gestures are indicative of adelirium state. As a non-liming example, the system will execute theCAM-ICU test by outputting multiple-choice questions and the patientwill communicate his answers by blinking.

As another non-liming example, the system will execute a reorientationevaluation by outputting the current time, date and location and thepatient will respond by maintaining open eyes gesture for a defined timeinterval.

Also, as another exemplary and non-limiting embodiment, the patient maybe prompted by audio or visual output to select between several options,e.g. “UP” (namely upwards gesture) for one selection, “DOWN” foranother, etc. By further exemplary and non-limiting embodiment, thepatient may be presented (e.g. through an audio output) with options andthereby prompting the patient to perform a gesture in a specific ornon-specific direction, make a series of blinks, close the eyelids for adefined period, etc., when the specific choice is presented. The latteris useful, for example, for a quick selection of letters for writingtexts. This embodiment may also serve the patient in responding themedical questionnaire.

Yet according to another exemplary and non-limiting embodiment, saidmedical questionnaire is a pain scale, optionally selected from aNumeric Rating Scale, Stanford Pain Scale, Brief Pain Inventory,Wong-Baker Faces, Global Pain Scale, Visual Analog Scale, and McGillPain Index. An additional medical questionnaire according to the presentdisclosure is an air hunger or a breathing discomfort questionnaire.

In some embodiments the data processing sub-system is further operableto receiving and classifying one or more physiological parameters, andidentifying such said physiological parameters, or any combination orsaid gestures and physiological parameters, indicative of the cognitivestate of the patient.

By some embodiments the identification involved the likelihood of thepatient to manifest a cognitive state based on a combination of said eyegestures and physiological parameters. By some embodiments saidcombination is rapid eye movements with elevated heart rate.

By other embodiment, the cognitive state is identified based on a seriesof gestures accumulated over a time period at a scale of minutes, hours,days, or months.

The scale minutes encompasses up to 10, 20, 30, 60, 120, or 240 minutes.

The scale of days encompasses up to 1, 2, 3, 4, 5, 7, 14, 30, or 60days.

The scale of months encompasses 1, 2, 4, 6, 8, 12, or 24 months.

At time the cognitive state is assessed based on the spontaneous orongoing gestures performed by the patients, independent of an inputprovided by the system.

The term physiological parameters encompass any sample of aphysiological measurement that is any signal that may be acquired fromthe patients' body, including any signal acquired from the patients'neural, cardiac, somatosensory, vocal, and respiratory system as well asmotion of selected muscles. The physiological parameters may be recordedby any sensor utility or a measuring device, a microphone, spirometer,galvanic skin response (GSR) device, touch or pressure probes,electrodermal response probe (skin conductance probe),electroencephalography (EEG) device, electroencephalography (ECoG)device, electromyography (EMG), electrooculography (EOG), andelectrocardiogram.

By some embodiments the physiological parameters, or any combination ofsaid eye gesture and physiological parameters are indicative of acognitive state of a patient.

By some embodiments there is provided a patient monitoring systemcomprising a plurality of patient monitoring systems (or sub-systems).By some embodiments, each of said plurality of patient monitoringsystems monitors a different patient.

By some embodiments said system further comprising a centralizedprocessor being operable for receiving signals representative of saidcognitive states from each of said patient monitoring systems (orsub-systems) and classifying such signals according to one or moredefined criteria. By some embodiments the classification, carried out bythe centralized processor, is based on criteria defined by a physicianor caregiver. Said criteria may represent medical urgencyconsiderations, temporal considerations, spatial consideration and anycombination thereof.

According to a non-limiting example, a set of patients hospitalized atthe same department are being monitored by said plurality of patientmonitoring systems (or sub-systems, wherein each patient in beingmonitored by a sub-system), and the signals transmitted by said systemsare being classified by the centralized processor so that the nurse unitis receiving the alerts ranked based on their medical urgency.

By some embodiments, the signal in an integrated signal, reporting anawake state of at least 2, 4, 6, 8, 10, 20, 50, 100 patients.

By some embodiments, the signal in an integrated signal, reporting asleep state of at least 2, 4, 6, 8, 10, 20, 50, 100 patients.

The system according to some embodiments of the present disclosureinvolves a training period, measured as the time needed to a patient toexecute at least one signal using the device, of up to 5, 10, 20, 30,60, 120, and 360 minutes, optionally measured from the first time pointof setting up the system with respect to the patient, for example propermounting of the headset.

In some embodiments the data processing sub-system is further operableto receiving and classifying any additional data (physiological ornon-physiological). A non-limiting example is receiving and classifyingpatients' answers (e.g. via reactive eye gestures or otherwise) to amedical questionnaire (optionally said medical questionnaire is outputby the system), to obtain an accumulated result of a medicalquestionnaire. As an example the patient may response to a stress levelquestionnaire via the system, and the results of said questionnairewould be received and classified by the system, optionally in combinedwith any one or combination of the eye image data, physiological dataand additional data. Said additional data may be any one or combinationof populational data (such as epidemiological data), personal data(genetic predisposition), medical history data or a non-medical datasuch as socioeconomical background. Said additional data may betransmitted to the data processing sub-system directly or, as anon-limiting example, via a nurse unit. In some embodiments the dataprocessing sub-system is further operable to receiving and classifyingadditional data and identifying any additional data or combination ofsaid gestures with additional data, indicative of the cognitive state ofthe patient.

By some embodiments the additional data, or any combination of said eyegesture, physiological parameters and additional data are indicative ofa cognitive state of a patient.

In some embodiments the data processing sub-system receives andprocesses (for example by means of natural language processing) audiodata. For example, once the patient is asked a question by anotherperson, e.g. a caregiver, the data processing sub-system may receive andprocess the physician speech and propose to the patient a response basedon a contextual analysis of the speech of the other person and thelanguage of the patient, including instructions to the patient, in hisown language. This embodiment would allow a patient in a foreign countryto easily communicate with the local physical and caregiver.

According to a second of its aspects there is provided a method foridentifying a cognitive state of a patient, the method comprising (a)recording image data of at least one of patient's eyes;

(b) classifying said image data into gestures; (c) identifying suchgestures indicative of the cognitive state of the patient; and (d)transmitting a signal communicating said cognitive state to a remoteunit.

By some embodiments, the method further comprising proving an output tothe patient.

By some embodiments, the method further comprising (a) recordingreactive image data representative of a reactive eye movement thatfollows said output; (b) classifying said reactive image data intoreactive gestures; and (c) identifying such gestures indicative of thecognitive state of the patient.

By some embodiments, the method further comprising (a) receiving andclassifying one or more physiological parameters; and (b) identifyingsuch said gestures and physiological parameters, or any combinationthereof, indicative of the cognitive state of the patient.

By some embodiments there is provided a method for identifying acognitive state of a patient, the method comprising (a) recording eyeimages of a patient from a plurality of patient monitoring systems; (b)classifying said image data into gestures; (c) identifying such gesturesindicative of the cognitive state of the patient to obtain an identifiedcognitive state; (d) classifying said identified cognitive stateaccording to one or more defined criteria.

By some embodiments there is provided a method for an integrated patientmonitoring for identifying a cognitive state of a plurality of patients,the method comprising (a) recording eye images of each patient from aplurality of patient monitoring systems; (b) classifying said image datafrom each of said systems into gestures; (c) identifying such gesturesindicative of the cognitive state of each patient to obtain anidentified cognitive states; (d) classifying said identified cognitivestates according to one or more defined criteria and (e) transmitting anintegrated signal communicating said cognitive states to a remote unit.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to better understand the subject matter that is disclosedherein and to exemplify how it may be carried out in practice,embodiments will now be described, by way of non-limiting example only,with reference to the accompanying drawings, in which:

FIG. 1 is a schematic block diagrams of a system in accordance anon-limiting embodiment of this disclosure.

FIG. 2 is a schematic block diagram of a system in accordance withanother non-limiting embodiment of this disclosure.

FIG. 3 displays an exemplary embodiment of the system.

FIG. 4 displays an exemplary embodiment of the system as may be worn bya patient.

FIG. 5 shows an exemplary embodiment of a high-level systemarchitecture.

FIG. 6 shows an exemplary embodiment of a display screen, or dashboard,of a remote unit being a medical staff station

DETAILED DESCRIPTION OF EMBODIMENTS

Reference is first made to FIG. 1 illustrating a schematic block diagramof a patient monitoring system in accordance with a non-limitingembodiment of this disclosure. The patient monitoring system 100comprises a camera 104, mounted on a head unit 102 configured forfitting onto a patient's head. The camera may also be mounted on anyfixed frame in the vicinity of the patient. The camera 104 is operablefor continuously capturing images of one or both of the patient's eyeand eyelid and generating image data representative thereof. The system100 includes a parallel or distributed data processing sub-system 106that is in data communication with the camera 104. The data processingsub-system 106 receive and process eye image data from said camera,

classify said eye image data into gestures and identify such gesturesindicative of the cognitive state of the patient. Next the dataprocessing sub-system 106 transmit a signal communicating said cognitivestate to a remote unit.

As a non-limiting example, an ICU patient hospitalized unconscious forseveral days, is being monitored by the patient monitoring system 100. Acaregiver had placed the wearable head unit 102 onto the patient's head.Once the patient opens his eyes for the first time, his eyes movement iscaptured by the camera 104. The image data from said camera 104 isreceived by the data processing sub-system 106. Next the image data isclassified into gestures, and in case an eye-opening gesture isclassified, a wakeful state is indicated and a “call for help” signal istransmitted wirelessly to the nearest nurse unit.

FIG. 2 shows a block diagram of the system of the present disclosure,wherein the system further includes an actuator module 108 that drivesthe first output unit 110. Output unit 110 may be a visual display, e.g.digital screen, or an audible device, e.g. speaker, headphones, etc.

As a non-limiting example, a patient hospitalized and suspected tosuffer from delirium is wearing the head unit 102 and is being monitoredby the patient monitoring system 100. Once the patients blink twice, hiseyes movement is captured by camera 104 and classified as a sequence oftwo blinks gesture by the data processing sub-system 106. The saidgesture initiates an output of a digital CAM-ICU medical questionnairevia the output unit 110 that is driven by the actuator module 108. Thepatients respond to the CAM-ICU evaluation by performing a reactive eyemovement, captured by the camera 104, and classified into reactive eyegestures via the data processing sub-system 106. These reactive eyegestures are taken to indicate whether the patient is indeed in adelirium state. If indeed a delirium state is identified by the dataprocessing sub-system 106 a delirium signal is transmitted to thepatient's physician by said data processing sub-system 106.

FIG. 3 and FIG. 4 show a non-limiting exemplary element of the system ofthis disclosure. The elements in FIG. 3 and FIG. 4 are assigned withnumerical indices that are shifted by 100 from the indices of theelements shown by FIG. 1 . For example, head unit indicted 102 in FIG. 1is indicated by 202 in FIG. 3 . Accordingly, the reader may refer to theabove text for details regarding the functionally of these elements.

FIG. 3 shows a non-limiting example of the system, including a head unit202, a camera 204 and a non-distributed data processing sub-system 206,which is in wireless communication (e.g. Wi-Fi, Bluetooth) with a remoteunit.

FIG. 4 shows a non-limiting illustration of the system as may be worn bya potential patient.

FIG. 5 shows a high-level system architecture according to anon-limiting exemplary embodiment. According the exemplary system, thesystem comprises a data processing sub-system 306 and a headset 302. Thesystem is in remote bi-directional communication with (i) a medicalstaff server 308 (for example via IoT protocol) which communicates witha remote medical staff station 310 (ii) a device settings cloud server312, via wi-fi communication and (iii) web-based additional applications314 via Bluetooth communication. The medical staff server comprises asystem database 316, an event scheduler 318 (for example for allowingthe medical staff to schedule a calendar event for a specific patientsuch as a “good morning” greeting every day at 08:00 or an initiation ofthe CAM-ICU or other medical questionnaire every 12 hours), a server 320for storing and retrieving data (such as the media files for the vocalmenu, a World Wide Web (WWW) page 322, and a text-to-speech applicationprogramming interface (API) 324. The staff server receives data via aremote family portal 326, for example voice messages. The family portal326 may transmit recommendations to the device setting cloud serverwhich comprise a web portal 328. The device settings cloud servercomprises a voice banking 330 (generate an original content usingsynthetic voice based on a previously recorded voice), text-to-speech332, and translation 334 APIs as well as a user database 336.

The system comprises an output unit and an actuator module for drivingan output selected from questionnaires, sets of audible questions andanswers, orientation messages (location, date, time), music, recordingsof family members etc. The outputs may be triggered by eye gesturesclassified by the system. In addition, response to said outputs may beprovided by the patient via a reactive gesture classified by the dataprocessing sub-system. A response may include answering questions.Overall, the system allows more natural, relaxed and controlledenvironment for the patient, thereby improving the quality of thehospitalization and reducing negative emotions during hospitalizationsuch as feeling lack of control, anxiety, fear of not being able tocommunicate and more. The system is also linked to a secured remotesupport cloud 338 enabling a reverse tunnel to a remote technicianservice.

FIG. 6 shows an example of a possible screen display (a dashboard) in amedical staff station which shows:

-   -   Communication log including answers of the patient to questions        sent from station in the form of audio (via speech).    -   Communication module presenting communication options to the        medical staff    -   Sleeping/awake pattern of the patient    -   Log of total alerts and alerts which require physical        intervention by the staff in the user room (user call for help        alert, camera is dislocated, device is disconnected from network        etc.)    -   Device location within the medical department reminder    -   Activity log of the user and device    -   Medical assessment questionnaire results (such as CAM-ICU, pain        scale and more)    -   Music, orientation, and family voice recordings buttons

1. A patient monitoring system for identifying a cognitive state of apatient, the system comprising a camera unit configured for recordingimages of an eye of the patient; a data processing sub-system in datacommunication with the camera and being operable to i. receive andprocess eye image data from said camera; ii. classify said eye imagedata into gestures and identify such gestures being indicative of thecognitive state of the patient, and iii. transmit a signal communicatingsaid cognitive state to a remote unit; an actuator module; and an outputunit; wherein said actuator module is configured to drive said outputunit to present an output to the patient; and wherein said dataprocessing sub-system records reactive image data representative of areactive eye movement that follows said output, classify said reactiveimage data into reactive gestures and identifying such gestures beingindicative of the cognitive state of the patient.
 2. The system of claim1, wherein the camera is carried on a head unit configured for fittingonto a patient's head.
 3. The patient monitoring system of claim 1,wherein the cognitive state is selected from wakefulness, delirium,cognitive decline, confusion, disorientation, abnormal attention,consciousness, pain, and depression.
 4. The patient monitoring of claim1, wherein the remote unit is an alert unit of an intensive care unit, anurse unit, or a device carried by a caregiver.
 5. The patientmonitoring system of claim 1, wherein the gesture is selected fromopening of at least one eye, closing at least one eye, pupil position,sequence of pupil positions, and sequences of eyelid blinks.
 6. Thepatient monitoring system of claim 1, wherein the opening of at leastone eye gesture is indicative of a wakeful state, and wherein an alertsignal is transmitted to a nurse unit.
 7. The patient monitoring systemof claim 1, wherein the signal to a remote unit is transmitted viawireless communication.
 8. (canceled)
 9. The patient monitoring systemof claim 1, wherein said reactive gestures are indicative of thecognitive state of the patient.
 10. The patient monitoring system ofclaim 9 wherein said output is an automatic medical questionnaire. 11.The patient monitoring system of claim 10, wherein said medicalquestionnaire is Confusion Assessment Method for the ICU (CAM-ICU). 12.The patient monitoring system of claim 10, wherein said medicalquestionnaire is a pain scale.
 13. The patient monitoring system ofclaim 10, wherein said medical questionnaire is an air hunger orbreathing discomfort questionnaire.
 14. The patient monitoring system ofclaim 10, wherein the opening of at least one eye initiates the CAM-ICU.15. The patient monitoring system of claim 1, wherein the dataprocessing sub-system is further operable to receiving and classifyingone or more physiological parameters, and identifying such saidphysiological parameters, or any combination of said gestures andphysiological parameters, indicative of the cognitive state of thepatient.
 16. A patient monitoring system comprising a plurality ofpatient monitoring systems of claim
 1. 17. The system of claim 16,further comprising a centralized processor being operable for receivingsignals representative of said cognitive states from each of saidpatient monitoring systems and classifying such signals according to oneor more defined criteria.
 18. A patient monitoring method foridentifying a cognitive state of a patient, the method comprising a.recording image data of at least one of patient's eyes; b. classifyingsaid image data into gestures; c. identifying such gestures that areindicative of the cognitive state of the patient; d. transmitting asignal communicating said cognitive state to a remote unit; e. providingan output to the patient; f. recording reactive image datarepresentative of a reactive eye movement that follows said output; g.classifying said reactive image data into reactive gestures; and h.identifying such gestures indicative of the cognitive state of thepatient.
 19. (canceled)
 20. (canceled)
 21. The method of claim 18,further comprising a. receiving and classifying one or morephysiological parameters; and b. identifying such of said gestures andphysiological parameters, or any combination thereof, indicative of thecognitive state of the patient.
 22. (canceled)
 23. A patient monitoringsystem for identifying a cognitive state of a patient, the systemcomprising a camera unit configured for recording images of an eye ofthe patient; a data processing sub-system in data communication with thecamera and being operable to i. receive and process eye image data fromsaid camera; ii. classify said eye image data into gestures and identifysuch gestures being indicative of the delirium state of the patientbased on defined criteria, and iii. transmit a signal communicating saidcognitive state to a remote unit; an actuator module; and an outputunit; wherein said actuator module is configured to drive said outputunit to present an output to the patient; wherein said output istriggered by eye gestures classified by the system and comprisesgeneric, or personally generated auditory content; wherein said dataprocessing sub-system records reactive image data representative of areactive eye movement that follows said output, classify said reactiveimage data into reactive gestures and identifying such gestures beingindicative of the delirium state of the patient based on definedcriteria.
 24. The patient monitoring system of claim 23, wherein saidoutput is selected from at least one of: questionnaires, sets of audiblequestions and answers, orientation messages, music, recordings of familymembers, instructions how to respond using his eye gestures.